Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implements PDF.js as a primary PDF extraction library #167

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

orfin
Copy link

@orfin orfin commented Aug 1, 2018

This PR makes PDF.js to be used as a default PDF converting library. That solves #138 and allows to run the library e.g. on AWS Lambda or environments that we don't want to pollute with external binaries.

It was written as a drop-in replacement for pdftotext, so all previous unit tests are passing -- however that means inside PDF.js we have funky "pdftotextOptions" assignments.

You are still able to use pdftotext when providing application/pdf-pdftotext as a typeOverride parameter - @dbashford we were thinking about the best way to have possibility to keep both libraries and chose that way. Maybe you see another way to do it more properly? :-)

Performance comparison

@dbashford you were concerned about performance of PDF.js as it's purely javascript based.
We made comparison between pdf.js and pdftotext and it seems that PDF.js may be slightly slower, however the difference is not huge.

Small PDF

$ time ./bin/textract test/files/testpdf-multiline.pdf
This is a test,
A multi-line test,
Lets hope it works

real	0m1.093s
user	0m0.965s
sys	0m0.122s
$ time ./bin/textract test/files/testpdf-multiline.pdf --typeOverride=application/pdf-pdftotext
This is a test,
A multi-line test,
Lets hope it works

real	0m0.891s
user	0m0.766s
sys	0m0.113s

Bigger PDF

$ time ./bin/textract test/files/two_columns.pdf

real	0m1.556s
user	0m1.673s
sys	0m0.145s
$ time ./bin/textract test/files/two_columns.pdf --typeOverride=application/pdf-pdftotext
real	0m0.988s
user	0m0.854s
sys	0m0.121s

@dbashford
Copy link
Owner

This is awesome. Literally 10 minutes before I saw the email for this I was thinking "I really need to spend some time looking into pdf.js".

I'll take a hard look at this, get this merged, touch up a few things and hopefully get it out with some spare time I have later today.

Thanks!!

@orfin
Copy link
Author

orfin commented Aug 1, 2018

Great to hear that and looking forward to your comments :-)

BTW. I'll be also working on implementing js-native extraction for DOC and PPT documents, potentially using following libraries:

However, I saw that ppt code is already in textract, but it's commented: https://github.com/dbashford/textract/blob/master/lib/extractors/ppt.js
what's the reason? ppt lib didn't work well?

@dbashford
Copy link
Owner

dbashford commented Aug 1, 2018

Yeah, the ppt library didn't work out from the beginning. 4 years ago I gave it a shot and that library just hasn't been updated. I kept the code in there to remind myself I gave it a shot.

@dbashford
Copy link
Owner

dbashford commented Aug 3, 2018

Heads up, trying to get a smaller bug fix release out before I include this as I suspect including this will require a major bump in version. If it works, not sure I really care to continue to support pdftotext, can push to 3.0 and leave it behind. Before something that drastic, though, want to get a few tiny things done and get a point release out.

(Hence the merge issues, apologies!)

@ghost
Copy link

ghost commented Jun 20, 2019

Is there an update on this? Is there a fork that uses pdf.js ?

@defi-bear
Copy link

https://github.com/Shing-Ho/textract
use this one. This doesn't require pdftotext and antiword

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants